Papers with Vernacular English
Exploring the Robustness of Task-oriented Dialogue Systems for Colloquial German Varieties (2024.eacl-long)
Copied to clipboard
| Challenge: | Mainstream cross-lingual task-oriented dialogue systems often overlook the transfer to lower-resource colloquial varieties due to limited test data. |
| Approach: | They propose to train a model for intent recognition and slot-filling in English and apply it to other languages. |
| Outcome: | The proposed model performs better than existing models on English and other languages. |
VALUE: Understanding Dialect Disparity in NLU (2022.acl-long)
Copied to clipboard
| Challenge: | English Natural Language Understanding systems outperform humans on benchmarks like GLUE and SuperGLUE, but they only use textbook Standard American English (SAE) . fewer studies have considered the effects of dialectal differences on performance . |
| Approach: | They propose a benchmark to evaluate the performance of English natural language understanding systems using a set of lexical and morphosyntactic transformation rules. |
| Outcome: | The proposed model outperforms humans on GLUE and SuperGLUE, but only on standard American English . the proposed model recruits fluent speakers of African American vernacular english to validate each feature transformation . |
It’s Morphin’ Time! Combating Linguistic Discrimination with Inflectional Perturbations (2020.acl-main)
Copied to clipboard
| Challenge: | Existing work on societal bias in NLP focuses on race and gender . linguistic background is a unique attribute that has been largely ignored in the field . |
| Approach: | They examine linguistic background to craft plausible adversarial examples that expose biases in popular NLP models. |
| Outcome: | The proposed model improves robustness without sacrificing performance on clean data. |
Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks (2025.acl-long)
Copied to clipboard
Fangru Lin, Shaoguang Mao, Emanuele La Malfa, Valentin Hofmann, Adrian de Wynter, Xun Wang, Si-Qing Chen, Michael J. Wooldridge, Janet B. Pierrehumbert, Furu Wei
| Challenge: | a study aims to assess the fairness and robustness of Large Language Models in dialectal queries . speakers of "non-standard" dialects are known to experience implicit and explicit discrimination . |
| Approach: | They propose to use a benchmark to assess the fairness of large language models in dialects . they hire speakers with computer science backgrounds to rewrite seven popular benchmarks based on AAVE . |
| Outcome: | The proposed benchmarks show that most models show significant brittleness and unfairness to queries in AAVE. |
Mind Your Inflections! Improving NLP for Non-Standard Englishes with Base-Inflection Encoding (2020.emnlp-main)
Copied to clipboard
| Challenge: | Inflectional variation is a common feature of World Englishes such as Colloquial Singapore English and African American Vernacular English. |
| Approach: | They propose a method to tokenize English text by reducing inflected words to their base forms before reinjecting the grammatical information as special symbols. |
| Outcome: | The proposed method defends against inflectional adversaries while maintaining performance on clean data. |
Investigating African-American Vernacular English in Transformer-Based Text Generation (2020.emnlp-main)
Copied to clipboard
Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, William Yang Wang
| Challenge: | Recent work in Natural Language Generation (NLG) uses a Transformer-based language model to generate high-quality, coherent text when prompted by arbitrary input. |
| Approach: | They evaluate the performance of a Transformer-based model that generates high-quality, coherent text when prompted by arbitrary input. |
| Outcome: | The proposed model improves on AAVE and SAE text with pretrained sentiment classifiers. |
Modeling Gender and Dialect Bias in Automatic Speech Recognition (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Dialect and gender-based biases have become an area of concern in language-dependent AI systems. |
| Approach: | They construct a podcast audio dataset and evaluate its performance . they then refine the models to better understand how finetuning can impact performance. |
| Outcome: | The proposed model improves on 13 hours of podcast audio transcribed by speakers of four US-based English dialects. |
Lost in Simulation: LLM-Simulated Users are Unreliable Proxies for Human Users in Agentic Evaluations (2026.acl-long)
Copied to clipboard
| Challenge: | Agentic benchmarks rely on LLM-simulated users to evaluate agent performance . however, the robustness, validity, and fairness of this approach remain unexamined . |
| Approach: | They investigate whether LLM-simulated users are reliable proxies for real human users . they find that agent success rates vary up to 9 percentage points across different LLMs . |
| Outcome: | The results show that simulated users underestimate success on challenging tasks while miscalibrate performance on moderately difficult tasks. |